docs(evals): update trace-level evaluator upgrade guide#3323
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…guide - Document deprecated set_trace_io/setActiveTraceIO bridge for keeping legacy trace-level evaluators running during the transition - Add AvailabilityBanner and self-hosted scoping with GitHub Discussion link - Add frontmatter description, Fast Preview terminology, Is Root Observation filter name, Langfuse Assistant link, remove stale last-updated footer - Add explicit anchor to the linked LLM-as-a-Judge setup heading Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…line - Remove the legacy "Traces (Legacy)" tab from the LLM-as-a-Judge setup guide; observation-level is now the only live-data path - Add deprecation note: legacy trace-level evaluators sunset by end of October 2026, multi-span evals build on the observation-centric model - Add the deprecation date to the Fast Preview rollout timeline on /docs/v4 with an explicit #rollout-timeline anchor Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dosubot
Bot
added
size:L
|
@claude review |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR is mostly a well-scoped documentation update, but it does not build: the new AvailabilityBanner on the FAQ page passes selfHosted: "v4", which isn't a valid availability id and throws at render time (the Vercel deployment already shows an Error status consistent with this). Please fix that value before merging; deferring for human review.
Extended reasoning...
Overview
The PR rewrites the trace-level → observation-level evaluator upgrade guide across three docs pages (llm-as-a-judge.mdx, v4.mdx, and the FAQ migration guide), adds an anchor to an existing heading, and restructures content into collapsible reference sections. All changes are content/documentation only — no application logic, auth, or data-handling code is touched.
Security risks
None. This is a documentation-only change with no new code paths, inputs, or data handling.
Level of scrutiny
This would normally warrant light scrutiny as a docs-only change, but the PR introduces a genuine build-breaking bug: content/faq/all/llm-as-a-judge-migration.mdx passes selfHosted: "v4" to <AvailabilityBanner />, an id not present in the component's fixed availabilities list (ee, team-add-on, full", "private-beta", "public-beta", "not-available"). This causes plan.availabilityto beundefinedand the render to throwCannot read properties of undefined (reading 'label'), which will fail static generation for this page during pnpm build`. This is already corroborated by the timeline: the Vercel deployment comment on this PR shows deployment status "Error". The PR description itself notes the full build/link checks were not run, which is consistent with this being missed.
Other factors
This is a real, build-breaking defect (already flagged as an inline finding), not a stylistic nit, so I'm deferring rather than approving despite the otherwise low-risk nature of the content changes.
- Clarify the usage of deprecated trace I/O methods in the v4 documentation, specifying the exact method names for both Python and JS/TS. - Enhance the migration guide to ensure users are aware of the transition from trace-level to observation-level evaluators and the necessary steps to maintain functionality during the upgrade. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Ready to merge, just waiting for Max PR. |
- Replace the hard "end of October 2026" date with "deadline will be announced" in the v4.mdx timeline entry and the llm-as-a-judge callout, matching the v4 launch messaging. - Rename "Fast Preview" to "Langfuse v4" and "legacy" to "deprecated" / "existing trace-level" in the new prose (FAQ description, intro, callout, rollback text, checklist, Getting help). - Remove em dashes from the PR-added callouts. - Fix the broken instrumentation anchor (#add-attributes-to-observations -> #add-attributes). - Run Prettier on v4.mdx (fixes the failing format CI check). The preceding merge of origin/main fixes the FAQ page crashing on AvailabilityBanner's selfHosted "v4" value, which only exists on newer main. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…om/langfuse/langfuse-docs into codex/trace-level-eval-deprecation
dosubot
Bot
added
size:XL
dosubot
Bot
added
size:L
2 participants
Summary
Verification
pnpm run format:checknode scripts/check-h1-headings.jsgit diff --checkFull build and link checks were not run for this focused documentation-only change.
Greptile Summary
This PR updates the trace-level evaluator upgrade documentation to reflect the v4 data model and the deprecation timeline (end of October 2026). The "Traces (Legacy)" UI tab is removed from the main LLM-as-a-Judge setup guide and replaced with a deprecation callout; the migration FAQ is substantially rewritten around the new agentic (Langfuse Assistant / coding assistant) upgrade flow.
llm-as-a-judge.mdx: Removes the "Traces (Legacy)" configuration tab and adds a deprecation warning callout with a deep link to the v4 rollout timeline; the matching explicit anchor is added to both this heading and thev4.mdxrollout-timeline heading.llm-as-a-judge-migration.mdx: Full rewrite — restructures the guide into three steps (SDK upgrade → Langfuse UI upgrade → review/validate), documents the in-app vs. coding-assistant paths, clarifies the score-cardinality change (one score per matching observation instead of per trace), and moves the before/after config example into a collapsible technical reference.v4.mdx: Adds a trace-level evaluator deprecation entry to the rollout timeline table and an explicit#rollout-timelineanchor so incoming deep links from the callout resolve correctly.Confidence Score: 4/5
Documentation-only change that is safe to merge; one method-name discrepancy between the migration guide and v4.mdx should be resolved before the page goes live to avoid confusing users attempting the transition.
The rewrite is clear and the cross-file anchor links are consistent. The one concrete problem is the mismatch between the deprecated method names used in the new migration guide (
set_current_trace_io()/setActiveTraceIO()) and those referenced in the existing v4.mdx (set_trace_io). Users following both pages simultaneously during an active upgrade will encounter conflicting names, and calling the wrong one will fail silently or raise a runtime error. Everything else — flow, callout placement, score-cardinality explanation, checklist — reads correctly.content/faq/all/llm-as-a-judge-migration.mdx(line 46) andcontent/docs/v4.mdx(line 143) need to be reconciled on the canonical deprecated method names for both the Python and JS/TS SDKs.Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[User has trace-level evaluators] --> B[Step 1: Upgrade SDK or ingestion path] B --> B1[Python SDK v4.7.0+] B --> B2[JS/TS SDK v5.4.0+] B --> B3[OTel: x-langfuse-ingestion-version: 4] B1 & B2 & B3 --> C{Keep legacy evaluator running during transition?} C -->|Yes — trace IO now empty| D[Use deprecated set_current_trace_io or setActiveTraceIO temporarily] C -->|No legacy trace IO used| E[Step 2: Open Evaluation screen in Langfuse UI] D --> E E --> F{Recommended upgrade path} F -->|Minimal config changes| G[In-app Langfuse Assistant upgrade] F -->|Broader instrumentation changes| H[Coding assistant upgrade] G & H --> I[Step 3: Review proposed config changes] I --> J[Create observation-level evaluators] J --> K{Score cardinality change?} K -->|Multiple observations match filter| L[Narrow filters using name/type/Is Root Observation] K -->|Single observation matches| M[Validate new evaluator results] L --> M M --> N{Keep legacy evaluator active temporarily?} N -->|Yes| O[Compare scores in parallel, then deactivate] N -->|No| P[Deactivate legacy evaluator immediately] O & P --> Q[Upgrade complete]%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% flowchart TD A[User has trace-level evaluators] --> B[Step 1: Upgrade SDK or ingestion path] B --> B1[Python SDK v4.7.0+] B --> B2[JS/TS SDK v5.4.0+] B --> B3[OTel: x-langfuse-ingestion-version: 4] B1 & B2 & B3 --> C{Keep legacy evaluator running during transition?} C -->|Yes — trace IO now empty| D[Use deprecated set_current_trace_io or setActiveTraceIO temporarily] C -->|No legacy trace IO used| E[Step 2: Open Evaluation screen in Langfuse UI] D --> E E --> F{Recommended upgrade path} F -->|Minimal config changes| G[In-app Langfuse Assistant upgrade] F -->|Broader instrumentation changes| H[Coding assistant upgrade] G & H --> I[Step 3: Review proposed config changes] I --> J[Create observation-level evaluators] J --> K{Score cardinality change?} K -->|Multiple observations match filter| L[Narrow filters using name/type/Is Root Observation] K -->|Single observation matches| M[Validate new evaluator results] L --> M M --> N{Keep legacy evaluator active temporarily?} N -->|Yes| O[Compare scores in parallel, then deactivate] N -->|No| P[Deactivate legacy evaluator immediately] O & P --> Q[Upgrade complete]Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "chore: push" | Re-trigger Greptile